1. Field of the Invention
The present invention relates to methods and systems for controlling delivery of digital messages such as E-mails transmitted via networks of computer systems.
2. Description of the Related Art
Transmission of information in the form of digital messages over computer communication networks such as e-mail and text messages is an essential component of modern lives, in both business and leisure. Many businesses derive income from services built “on top of” digital messaging. For example, almost all online purchases trigger a message to confirm that a monetary transaction has taken place. Schools are increasingly making use of alerting services that broadcast text messages to mobile devices. Businesses use E-mail and instant messaging as an essential part of day-to-day business communication, both internally and externally. Internet Service Providers (“ISPs”) enable email services for personal and business use. E-mail Service Providers (“ESPs”) enable other businesses to efficiently send their marketing and other types of communication without requiring a significant investment in sending infrastructure. “Aggregators” provide gateways to text messaging protocols and act as an endpoint for submitting Short Message Service (“SMS”) and Multimedia Message Service (“MMS”) messages to cell phone provider networks.
Many digital messaging systems that send digital messages over the Internet use protocols such as Simple Mail Transfer Protocol (“SMTP”), in the case of E-mail, and other protocols when the digital message is other than e-mail, to send digital messages from one server to another. The messages can then be retrieved with a client using services such as Post Office Protocol (“POP”) or Internet Message Access Protocol (“IMAP”). Other protocols for sending digital messages that are e-mails include, but are not limited to, POP3, X.400 International Telecommunication Union standard (X.400), and the Novell message handling service (“MHS”) as well as the Extended Simple Mail Transfer Protocol (“ESMTP”). Examples of prior art message delivery systems are illustrated in FIGS. 1A, 1B, 1C and 2.
SMTP transports digital messages among different hosts within the Transmission Control Protocol/Internet Protocol (“TCP/IP”). Under SMTP, a client SMTP process opens a TCP connection to a server SMTP process on a remote host and attempts to send mail across the connection. The server SMTP process listens for a TCP connection on a selected port and the client SMTP process initiates a connection on the selected port. When the TCP connection is successful, the two processes execute a simple request-response dialogue, defined by the SMTP protocol, during which the client process transmits the mail addresses of the originator and the recipient(s) for a message. When the server process accepts these mail addresses, the client process transmits the e-mail instant message. The e-mail message contains a message header and message text (“body”) formatted in accordance with RFC 822 STD 11, the Standard for the format of Internet Text Messages. Mail that arrives via SMTP is forwarded to a remote server or it is delivered to mailboxes on the local server. Similar protocols are used for other digital message formats (i.e., other than e-mail).
The SMTP protocol (see, e.g., RFC 821 or the recently updated standard RFC5321) supports both end-to-end (no intermediate message transfer agents “MTAs”) and store-and-forward mail delivery methods. The end-to-end method is used between organizations, and the store-and-forward method is chosen for operating within organizations that have TCP/IP and SMTP-based networks. A SMTP client will contact the destination host's SMTP server directly to deliver the mail. It will keep the mail item from being transmitted until it has been successfully copied to the recipient's SMTP.
This is different from the store-and-forward principle that is common in many other electronic mailing systems, where the mail item may pass through a number of intermediate hosts in the same network on its way to the destination and where successful transmission from the sender only indicates that the mail item has reached the first intermediate hop. The RFC 821 standard defines a client-server protocol. The client SMTP is the one which initiates the session (that is, the sending SMTP) and the server is the one that responds (the receiving SMTP) to the session request. Because the client SMTP frequently acts as a server for a user-mailing program, however, it is often simpler to refer to the client as the sender-SMTP and to the server as the receiver-SMTP. An SMTP-based process can transfer electronic mail to another process on the same network or to another network via a relay or gateway process accessible to both networks. An e-mail message may pass through a number of intermediate relay or gateway hosts on its path from a sender to a recipient.
An example of the components of an SMTP system is illustrated in FIG. 1A. Systems that send digital messages other than e-mail have similar components. Referring to FIG. 1A, users deal with a user agent (“UA”, e.g., 120 and 160). The user agents for Windows™ include Microsoft Outlook™. The exchange of e-mail using, for example, TCP, is performed by a Message Transfer Agent (“MTA” e.g., 140 or 150). The user's local MTA maintains a mail queue so that it can schedule repeat delivery attempts in case a remote server is unable. Also the local MTA delivers mail to mailboxes, and the information can be downloaded by the UA (see FIG. 1A). The SMTP protocol defines a mechanism of communication between two MTAs across a single TCP connection. As a result of a user mail request, the sender-SMTP establishes a two-way connection with a receiver-SMTP. The receiver-SMTP can be either the ultimate destination or an intermediate one (known as a mail gateway). The sender-SMTP will generate commands, which are replied to by the receiver-SMTP (see FIG. 1C).
A typical e-mail delivery process is as follows. Delivery processes for digital messages other than e-mail follow similar scenarios. In the following scenario, a sending user at terminal 100 sends an e-mail message to a recipient user at an e-mail address: recipient@example.org. Recipient can review that e-mail message at terminal 170. Recipient's ISP uses mail transfer agent MTA 150.
Initially, the sending user composes the message and actuates or chooses “send” using mail user agent (MUA) 120. The sending user's e-mail message identifies one or more intended recipients (e.g., destination e-mail addresses), a subject heading, and a message body, and may specify one or more attachments.
Next, MUA 120 queries a DNS server (not shown) for the IP address of the local mail server running MTA 140. The DNS server translates the domain name into an IP address of the local mail server and User agent 120 opens an SMTP connection to the local mail server running MTA 140. The message is transmitted to the local mail server using the SMTP protocol. An MTA (also called a mail server, or a mail exchange server in the context of the Domain Name System) is a computer program or software agent that transfers electronic mail messages from one computer to another. The message is transmitted to MTA 150 from MTA 140 using the SMTP protocol over a TCP connection.
The recipient user at terminal 170 has user agent 160 download recipient's new messages, including the sending user's e-mail message. The recipient's MTA 150 is responsible for queuing up messages and arranging for their distribution. MTA 150 “listens” for incoming e-mail messages on the SMTP port, and when an e-mail message is detected, it handles the message according to configuration settings or policies chosen by the system administrator. Those policies can be defined in accordance with relevant standards such as Request for Comment documents (RFCs). Typically, the mail server or MTA must temporarily store incoming and outgoing messages in a queue, known as a the “mail queue.” Actual size of the mail queue is highly dependent on one's system resources and daily message volumes.
In some instances, referring to FIG. 1B, communication between a sending host (client) and a receiving host (server), could involve a process known as “relaying”. In addition to one MTA at the sender site and one at the receiving site, other MTAs, acting as client or server, can relay the electronic mail across the network. This scenario of communication between the sender and the receiver can be accomplished through the use of a digital message gateway, which is a relay MTA that can receive digital messages prepared by a protocol other than SMTP and transform it to the SMTP format before sending it. The digital message gateway can also receive digital messages in the SMTP format, change it to another format, and then send it to the MTA of the client that does not use the TCP/IP protocol suite. In various implementations, there is the capability to exchange mail between the TCP/IP SMTP mailing system and the locally used mailing systems. These applications are called digital message gateways or mail bridges. Sending digital messages (e.g., e-mail) through a digital message gateway may alter the end-to-end delivery specification, because SMTP will only guarantee delivery to the mail-gateway host, not to the real destination host, which is located beyond the TCP/IP network. When a digital message gateway is used, the SMTP end-to-end transmission is host-to-gateway, gateway-to-host or gateway-to-gateway; the behavior beyond the gateway is not defined by SMTP.
Existing digital message sending systems are sometimes unsatisfactory. Digital messages (e.g., e-mail) sent to a given domain (e.g., a particular ISP) or site may be blocked. However, typically, the blocking domain will either not specify why the digital message was blocked or will provide incomplete information as to why the message was blocked. In one example, the reason why a domain will block incoming digital messages is that a digital message quota, where the quota is based for example on a number of allowed digital messages per unit of time (e.g., per day, per week, per month) from the sending domain has been exceeded. However, the conventional sending digital messaging sending system (e.g., MTA 140 of FIG. 1A) does not learn that a quota has been exceeded from the message failure notices that are returned to the sending MTA by the blocking domain. So the digital message sending system continues to resend the blocked digital messages and send any new additional digital messages (e.g., e-mail, text messages, etc.) in its queue to the blocking domain even though there is no chance that the blocking domain will accept the resent digital messages or the new digital messages from the digital message sending system because the quota has been exceeded.
In another example, the receiving domain may be down because of equipment failure or scheduled maintenance. There are, therefore, many reasons why a digital message sending system may send digital messages to the receiving domain and meet with failure. But the digital message sending system will continue to resend the digital message and any new digital message in its queue to the receiving domain on some predetermined retry basis without intelligently backing off from its efforts to send out the digital message to the domain that is down.
It is reported that the majority of E-mail traffic (in excess of 80-90%, depending on the source of the statistics) is classified as “Spam” (unsolicited or otherwise unwanted messages). It is increasingly difficult for both senders and receivers to ensure that E-mail messages arrive successfully at the intended recipient's inbox. This is due in part to a pessimistic stance being adopted by the receivers; their inbound systems are burdened with the task of accurately classifying and processing messages that may have Spam, Virus or Phishing content and E-mail receivers are fighting a constant battle with the increasing volume of unwanted messages and trying to balance that with an increasing cost in infrastructure to deal with that load. This has resulted in systems programmed to implement a set of incoming E-mail processing policies that penalize bulk senders by throttling back reception rates or delaying inbound mail reception, making it difficult for their E-mail to reach the inbox on-time if ever. These receiver policies result in “Deliverability Issues” for the senders; not only is delivery slowed down, but it is often done in such a way that makes it difficult for the sending system to take appropriate action.
Many smaller E-mail sending sites resort to tricks at the firewall that result in traffic mysteriously stopping, rather than sending back a definitive response code. This tactic is aimed at illegitimate senders where it is effective at avoiding load, but is especially harmful to legitimate senders because it increases the resource utilization of the sending system by making the sending system try harder to deliver the messages. The slowdown in reception causes a bottleneck on the sending system that increases resource utilization and CPU load, and can in turn have an impact on the sending rates for mail intended for other destinations.
If a sender is seen to trigger enough of these receiver policies, the sender may have their sending IP address placed on a “blacklist” shared with other recipient domains. Blacklisting causes the triggering sender to have deliverability issues at other receiving sites too. The result of these conditions is that it is difficult for a legitimate bulk sender to achieve high throughput without establishing some kind of relationship, either directly or indirectly through a “deliverability expert”, with the administrators for the domains to which they intend to deliver their mail. Such a relationship will enable the sender to determine traffic shaping parameters or rules that are acceptable to the recipient and are typically dependent on the volume of messages being sent over an extended period of time and the historical and ongoing reputation of the sender (e.g., is this sender a known spammer, or does this sender otherwise have a high complaint rate from the ultimate recipients?). Other factors may apply; the inbound mail policies are not necessarily public and, by necessity, the policies change over time.
Having established appropriate traffic shaping parameters or rules for a given recipient domain, the sender's software can be configured to respect those rules. As the rules change, the software configuration will also need to be adjusted accordingly. This is an ongoing support burden for both the sender and the receiver of messages. Even with the correct shaping rules in place, it is possible for a sender to trip up and damage their reputation.
Many senders re-sell their services to others and act as Email Service Providers (ESPs). It is possible for an ESP customer to send a batch of e-mail that is widely regarded as spam; if enough of this content is sent through the ESP system it can result in blacklisting the system and impact the mail throughput for other customers of that ESP. A diligent administrator at an ESP should be able to monitor and take action in the aforementioned case, but would need to be very focused on that task to be effective.
While much of this document refers to E-mail, the same principles and problems apply to all digital messaging protocols and platforms; text messaging (SMS), multimedia messaging (MMS), instant messaging (XMPP and other proprietary protocols). Popular social media platforms, such as Twitter.supSM and Facebook.supSM also expose messaging interfaces for external applications to similar problems.
There is a need, therefore, for a convenient, flexible and effective method and system for adapting to changes and controlling delivery of digital messages such as E-mails transmitted via networks of computer systems.